Standalone upgrade

Note Before proceeding with any upgrades, back up your Collibra DQ metastore.

Warning Rolling back Collibra DQ to an older version is not supported. Please contact Collibra Support with any questions.

Download the Collibra DQ Upgrade Package

Note Starting with the December 2021 release, all Collibra DQ customers upgrading or patching receive the Full package, not the Base package, and should follow the same upgrade steps described in this section.

Download the tarball using the signed link to the full package tarball provided by Collibra. Replace <signed-link-to-full-package> with the link provided.

### Go to the OWL_BASE (home directory of the user is most common)
### This example we will use /home/owldq installing as the user owldq

cd /home/owldq 

### Download & untar
curl -o dq-full-package.tar.gz "<signed-link-to-full-package>"
tar -xvf dq-full-package.tar.gz

### Clean-up unnecessary tarball (optional)
rm dq-full-package.tar.gz

Upgrade steps

Version < 2023.09 to version < 2023.09
Version < 2023.09 to version ≥ 2023.09
Version ≥ 2023.09 to version > 2023.09

This section provides steps to upgrade standalone deployments of Collibra DQ from a version older than 2023.09 to a different version older than 2023.09.

Copy the contents of the provided package, for example, owl-<new-version>-<SPARK301>-package-full.tar.gz to the system being upgraded (extract contents).
Tip Untar the contents into a uniquely named folder, for example, 2023-05-dq-upgrade.

Stop the Collibra DQ Web process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlweb

Stop the Collibra DQ Agent process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlagent

Move the old jars from the owl/bin folder.

Copy

mv owl-webapp-<oldversion>-<spark301>.jar /tmp
mv owl-agent-<oldversion>-<spark301>.jar /tmp
mv owl-core-<oldversion>-<spark301>.jar /tmp

Copy the new jars into the owl/bin folder from the extracted package.

Copy

mv owl-webapp-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv owl-agent-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv owl-core-<newversion>-<spark301>.jar /home/owldq/owl/bin

Copy the latest owlcheck and owlmanage.sh to the /opt/owl/bin directory.
Start the Collibra DQ Web application.
Copy
```
./owlmanage.sh start=owlweb
```
Start the Collibra DQ Agent.
Copy
```
./owlmanage.sh start=owlagent
```
Validate the number of active services.
Copy
```
ps -ef | grep owl
```

This section provides steps to upgrade standalone deployments of Collibra DQ from a version older than 2023.09 to version 2023.09 or newer.

Copy the contents of the provided package, for example, dq-<new-version>-<SPARK301>-package-full.tar.gz to the system being upgraded (extract contents).
Tip Untar the contents into a uniquely named folder, for example 2023-09-dq-upgrade.

Stop the Collibra DQ Web process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlweb

Stop the Collibra DQ Agent process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlagent

Move the old jars from the owl/bin folder.

Copy

mv owl-webapp-<oldversion>-<spark301>.jar /tmp
mv owl-agent-<oldversion>-<spark301>.jar /tmp
mv owl-core-<oldversion>-<spark301>.jar /tmp

Copy the new jars into the owl/bin folder from the extracted package.

Copy

mv dq-webapp-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv dq-agent-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv dq-core-<newversion>-<spark301>.jar /home/owldq/owl/bin

Copy the latest owlcheck and owlmanage.sh to /opt/owl/bin directory.

Tip You may also need to run chmod +x owlcheck owlmanage.sh to add execute permission to owlcheck and owlmanage.sh.

Start the Collibra DQ Web application.
Copy
```
./owlmanage.sh start=owlweb
```
Start the Collibra DQ Agent.
Copy
```
./owlmanage.sh start=owlagent
```
Validate the number of active services.
Copy
```
ps -ef | grep owl
```

This section provides steps to upgrade standalone deployments of Collibra DQ from version 2023.09 or newer to a version newer than 2023.09.

Copy the contents of the provided package, for example, dq-<new-version>-<SPARK301>-package-full.tar.gz to the system being upgraded (extract contents).
Tip Untar the contents into a uniquely named folder, for example 2023-11-dq-upgrade.

Stop the Collibra DQ Web process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlweb

Stop the Collibra DQ Agent process.

Copy

cd /owlhome/owl/bin
./owlmanage.sh stop=owlagent

Move the old jars from the owl/bin folder.

Copy

mv dq-webapp-<oldversion>-<spark301>.jar /tmp
mv dq-agent-<oldversion>-<spark301>.jar /tmp
mv dq-core-<oldversion>-<spark301>.jar /tmp

Copy the new jars into the owl/bin folder from the extracted package.

Copy

mv dq-webapp-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv dq-agent-<newversion>-<spark301>.jar /home/owldq/owl/bin
mv dq-core-<newversion>-<spark301>.jar /home/owldq/owl/bin

Copy the latest owlcheck and owlmanage.sh to /opt/owl/bin directory.
Start the Collibra DQ Web application.
Copy
```
./owlmanage.sh start=owlweb
```
Start the Collibra DQ Agent.
Copy
```
./owlmanage.sh start=owlagent
```
Validate the number of active services.
Copy
```
ps -ef | grep owl
```

Additional Log4J notes

0.1 Place Log4j-1.2-api-2.17.1.jar (as of 2022.02) into /<install-home>/owl/spark/jars

Note For versions 2021.12 and 2022.01, the name of the package is Log4j-1.2-api-2.17.0.jar.

Action For all Collibra DQ customers, particularly those leveraging CLI mode.

Go to the same folder where you extracted the Collibra-provided upgrade package.
Go to <location_of_2022-02-dq-upgrade>/packages/install-packages.

Extract log4j-1.2-api-2.17.1.jar with the following command:

tar -xvf spark-extras.tar.gz spark-extras/log4j-1.2-api-2.17.1.jar

Move the log4j-1.2-api-2.17.1.jar file to the /<install-path>/spark/jars folder.

FAQ

Q: (When) do I need to move log4j-1.2-api-2.17.1.jar before or after swapping the main Collibra DQ jars?

A: The sequence does not matter.

Q: (What) if I don't follow these additional upgrade steps?

A: If your SPARK_SUBMIT_MODE within owl-env.sh is set to SPARK_SUBMIT_MODE=native, Collibra DQ will function properly without the above additional upgrade step, with the exception of CLI mode.

0.2 Remove a legacy properties file

Action Only for Collibra DQ customers upgrading Agents installed on Cloudera CDP Hadoop Edge Nodes.

Go to /<agenthome>/owl/config/.
Remove the log4j-cluster.properties file.

FAQ

Q: (When) do I need to remove log4j-cluster.properties before or after swapping the main Collibra DQ jars?

A: Remove the file before restarting owl-agent. Otherwise, stop owl-agent again, remove the file, then restart owl-agent.

Q: (What) if I don't follow these additional steps?

A:If you use agents on Hadoop edge nodes, you will receive errors when running DQ Jobs as a result of engaging a method that no longer exists.

Q: What should I do if I am not a vendor-supported Cloudera CDP version?

A: Our testing and guidance mainly applies to vendor-supported (non-EOL) Cloudera CDP versions. Other Hadoop variants may handle logging differently and may require the legacy properties file. In short, feel free to first upgrade without this step, then remove the log4j-cluster.properties file if DQ Jobs are running into issues.

Upgrading data source drivers

When new data source drivers are available, they are listed in the Release Notes or recommended to you directly by Collibra. Determine which drivers need to be updated and follow these steps:

Confirm with Collibra Support which drivers need to be updated.
From the previously extracted tarball provided to you by Collibra, locate the drivers.tar.gz file and extract the contents into a new directory called drivers.
Replace the drivers: Replace OWL_BASE/owl/drivers/<old-driver> with the new drivers extracted from the tarball OWL_BASE/owl/drivers/<new-driver>.
Example If you replace an old Databricks driver with a new one, the file path might look like OWL_BASE/owl/drivers/databricks.